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DESCRIPTION 

APPARATUS, SYSTEM, AND METHOD 
OF ELECTRONIC MAIL PROCESSING 

5 

Technical Filed 

The present invention relates to an apparatuses and a method of 
processing electronic mail (e-mail) by video or audio data, and more particularly 
to reproduction technologies of a reply mail citing video or audio data of the 
10 original. 

Background Art 

Recently, owing to popularity of information terminals using video 
and audio recording functions, electronic mails using video or audio as well as 

15 text (video mail and audio mail, respectively) are increasing aside. Generally, in 
exchange of electronic mails, a receiver of an electronic mail often sends a 
reply mail by citing a part or whole of the received mail and/or newly adding a 
text. In video or audio mail, a system of replying by citing a part or whole of the 
received video or audio mail is proposed, for example, in patent document 1. 

20 *** Patent document 1 : JP2000-209347 

Disclosure of Invention 

(Problems to be solved by the invention) 

In the system proposed in patent document 1, the receiver of the 

25 reply mail must reproduce the video or audio data once created by the own self 
which is cited in the reply mail, when reviewing the video or audio data sent 
from the partner in reproduction of the reply mail, and it takes much time in 
reproduction. In particular, since the receiver of the reply mail knows the 
content of the cited video or audio data once created by the own self, and new 

30 information is not obtained if the cited data is reproduced precisely, and it is a 
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mere waste of time. From such viewpoint, regarding the reply mail citing the 
video mail or audio mail, the reviewing efficiency thereof is poor for the user. 

(Means for solving the problems) 
5 The invention is devised in the light of the above problems, and it 

is hence an aim thereof to provide a processing apparatus and a processing 
method capable of reviewing the audio or video mail in the reply mail efficiently. 

An electronic mail processing apparatus according to the present 
invention includes a data receiving section operable to receive mail data 
10 including video data or audio data, a cited data detecting section operable to 
detect a cited part of the received video data or audio data as cited data from 
the mail data received by the data receiving section, and a reproduction control 
section operable to control a reproducing method of the received mail data. 
The reproduction control section controls the reproduction method so that a 
15 reproduction method for the cited data is deferent from a reproduction method 
for data other than the cited data in the received mail data. 

The reproduction control section may control the reproduction 
speed of the mail data so as to reproduce the data other than the cited data in 
the received mail data at a first speed, and reproduce the cited data at a second 
20 speed which is faster than the first speed. 

The cited data may include information relating to a creator of the 
mail data in a specified region. In this case, the cited data detecting section can 
detect the cited portion by using the information relating to the creator. 

The video data or audio data included in the cited data may be 
25 data compressed by a specified compressing method and having a header 
portion in which information relating to a creator of the mail data is recorded. 

The cited data may include information relating to a creator of the 
mail data in a form of a digital watermark embedded in the video data or audio 
data. In this case, the cited data detecting section can detect the cited data by 
30 using the information relating to the creator embedded in a form of a digital 
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watermark. 

The electronic mail processing apparatus may further include a 
data creating section operable to create the mail data for transmission including 
video data or audio data. In this case, when creating the mail data, the data 
5 creating section stores information relating to a creator of the mail data in a 
specified recording region of the video data or audio data. Alternatively, the 
data creating section may embed information relating to a creator of the mail 
data in a form of a digital watermark In the video data or audio data. 

An electronic mail processing method according to the present 
10 invention includes receiving mail data including video data or audio data, 
detecting a cited part of the video data or audio data as cited data from the 
received mail data, and reproducing the cited data detected from the received 
mail data and data other than the cited data in the received mail data in different 
methods. 

15 An electronic mail processing system according to the present 

invention includes a first mall processing apparatus and a second mail 
processing apparatus which exchanges electronic mails with the first mail 
processing apparatus. The first mail processing apparatus includes a data 
transmitting and receiving section operable to transmit and receive mail data 

20 including video data or audio data, a cited data detecting section operable to 
detect a cited part of video data or audio data as cited data from the received 
mail data, a reproduction control section operable to control a reproducing 
method of the received mail data so that a reproduction method for the cited 
data is deferent from a reproduction method for data other than the cited data in 

25 the received mail data, and a data creating section operable to create mall data 
for transmission by adding information relating to a creator of the mail data to 
video data or audio data included in the mail data. The second mail processing 
apparatus includes a data transmitting and receiving section operable to 
transmit and receive mail data, and a data editing section operable to edit the 

30 mail data received in the data transmitting and receiving section. The second 
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mail processing apparatus creates a reply mail including as cited data a part or 
whole of video data and audio data included in the received mail data, by the 
data editing section. Then it transmits a reply mail to a sender of the received 
mail data through the data transmitting and receiving section. 

5 

(Effects of the Invention) 

According to the Invention, when receiving and reviewing the reply 
mail citing the video or audio data created by the own user, it is possible to 
reproduce the cited portion and other portion in different reproduction methods. 
10 Hence, the cited portion can be reproduced at fast speed, while the new video 
or audio portion added by the sender can be reproduced at normal speed, so 
that the replied video or audio mail can be reviewed efficiently in a shorter time 
while understanding the correspondence between the original data and replied 
data. 

15 

Brief Description of Drawings 

Fig. 1 is a block diagram of mail processing system in a first 
embodiment of the invention. 

Fig. 2A is a diagram explaining a structure of a video mail or audio 

20 mail. 

Fig. 2B is a diagram explaining the reply mail citing a part of the 
video or audio data. 

Fig. 3 is a diagram showing a structure of video data added 
creator information. 

25 Fig. 4 is a diagram showing a structure of audio data added 

creator information. 

Figs. 5A through 5C are diagrams explaining a position of creator 
information added to the entire video or audio data. 

Fig. 6 is a block diagram of an e-mail processing system in a 
30 second embodiment of the invention. 
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Best Mode for Carrying Out the Invention 

Preferred embodiments of the invention are described below while 
referring to the accompanying drawings. 

5 

First Embodiment 

Fig. 1 is a block diagram of an e-mail (electronic mail) processing 
system in the first embodiment of the invention. The e-mail processing system 
includes an e-mail processing apparatus 100 and an e-mail replying apparatus 
10 200. 

The e-mail processing apparatus 100 is an apparatus for 
processing electronic mail expressed by video or audio (video mail or audio 
mail). The video mail or audio mail described here may include a mail which is 
composed of only video data or audio data, and a text mail to which video data 

15 or audio data is attached. 

The e-mail replying apparatus 200 has a function of receiving a 
mail transmitted from the e-mail processing apparatus 100, and creating and 
transmitting a reply mail to the received mail. The e-mail replying apparatus 
200 includes a data transmitting and receiving section 201, a data storage 

20 section 202, and an audio and video editing section 203. The data transmitting 
and receiving section 201 transmits an electronic mail to outside, and receives 
an electronic mail transmitted from outside. The data storage section 202, while 
cooperating with the data transmitting and receiving section 201, stores a part 
or whole of video data, audio data, data creator information, mail destination 

25 information, mail source information and user information of sent and received 
mails when transmitting and receiving mails. The audio and video editing 
section 203 creates video data or audio data newly for a reply mail by using a 
part or whole of the received video data and audio data stored in the data 
storage section 202. 

30 When the audio and video editing section 203 creates a reply mail 
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citing a part of the sent nnail data, the cited data is directly contained in the reply 
mail. Fig. 2A and Fig. 2B are diagrams explaining structures of the sent mail 
and reply mail. For example, when the mail transmitted from the e-mail 
processing apparatus 100 to the e-mail replying apparatus 200 contains parts A 
5 to D of video or audio data as shown in Fig. 2A, the audio and video editing 
section 203 can cite data A and data C in the sent mail and insert video data or 
audio data X, Y after each cited date to create a reply mail, as shown in Fig. 2B, 
The audio and video editing section 203 adds information (creator 
information) relating to a creator of the mail data to the video and audio data to 

10 be newly added in the reply mail. The creator information includes the user ID 
(identification information) of the reply mail creator, and device ID of the e-mail 
replying apparatus 200 and the other information. The details of the method of 
adding the creator information to the video and audio data is explained later. 

Returning to Fig. 1, the details of the e-mail processing apparatus 

15 100 is described. 

The e-mall processing apparatus 100 includes a data transmitting 
and receiving section 101, a data storage section 102, a cited data detecting 
section 103, an audio and video reproduction control section 104, an audio and 
video reproducing section 105, and a mail data creating section 106. The e- 

20 mail processing apparatus 100 holds user information 31. 

The data transmitting and receiving section 101 transmits an 
electronic mail from the e-mail processing apparatus 100 to outside, and 
receives an electronic mail transmitted to the e-mail processing apparatus 100. 
The mail data transmitted and received in the data transmitting and receiving 

25 section 101 includes a part or whole of video data, audio data, data creator 
information, mail destination information, mail source information and user 
information. 

The data storage section 102, while cooperating with the data 
transmitting and receiving section 101, stores the information included in the 
30 mail data transmitted and received in the data transmitting and receiving section 
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101 when transmitting and receiving mail. 

The mail data creating section 106 creates video and audio mail 
according to the user's operation. At this time, the mail data creating section 
106 adds the information (creator information) relating to the creator of the mail 
5 data to the video and audio data of the video and audio mail. The creator 
information includes the user ID (identification Information) of the mail creator, 
and device ID of the electronic mail transmitting processing apparatus 100 and 
other information. The creator information used in data creation of the sent mail 
is stored in a specified recording region. The details of a method of adding 

10 creator information to the video and audio data is explained later. 

The cited data detecting section 103 extracts the information 
(creator information) relating to the data creator from the data which is received 
in the data transmitting and receiving section 101 and recorded in the data 
storage section 1 02, and detects the data of the cited portion (cited data) from 

15 the received mail data by using the creator information. For example, in the 
case of Fig. 2B, as cited data, portions of data A and data C are detected. More 
specifically, the cited data detecting section 103 extracts the creator information 
from the received data, compares the extracted creator information with the 
creator information added when creating the cited original mail by the mail data 

20 creating section 106, and thereby detects the cited data. 

The audio and video reproduction control section 1 04 generates a 
control signal for reproducing the cited data detected by the cited data detecting 
section 103 at fast reproduction speed while reproducing other portion, that is, a 
portion newly added in the reply mail at normal reproduction speed, and sends 

25 the control signal to the audio and video reproducing section 105. 

The audio and video reproducing section 105 reproduces the 
video and audio data of the reply mail at the reproduction speed instructed by 
the control signal from the audio and video reproduction control section 104. 

According to this configuration, the receiver of the mail can review 

30 the reply mail citing the video and audio data of the original mail created by the 
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own receiver while the cited portion is reproduced at fast speed and the video 
and audio portion newly added by the sender is reproduced at normal 
reproduction speed, so that the receiver can review the returned video and 
audio mail efficiently while understanding the correlation of the original data and 
5 reply data. 

The video data or audio data included in the mail data may include 
not only a picture data, but also graphic data or synthetic data created by the 
computer and so on. 

Thus, in the embodiment, by referring to the creator information 
10 extracted from the mail data received, the cited data detecting section 103 of 
the e-mail processing apparatus 100 detects the cited data in the reply mail. 
The method of adding the creator information in the data is specifically 
explained below. 

Fig. 3 Is a diagram showing an example of structure of video data 

15 included in the electronic mail handled in the present embodiment. Fig. 4 is a 
diagram showing an example of structure of audio data included in the 
electronic mail handled in the present embodiment. As shown In these 
diagrams, data creator information 55 or 65 is added to header 51 or 61 of each 
GOP (group of pictures) 50 of video data or each audio frame 60 of audio data. 

20 By adding the creator information in each GOP unit or each audio 

frame unit, if edited in the GOP unit or audio frame unit on the reply mail editing 
side, that is, on the e-mail replying apparatus 200 side, the creator information 
of each GOP or audio frame is maintained. Hence, even when receiving video 
mail or audio mail edited and cited in the GOP unit or audio frame unit, the cited 

25 data detecting section 1 03 can correctly detect the cited portion by comparing 
the creator information in each GOP or audio frame with the user information. 

In the present embodiment, in the GOP or audio frame, the 
creator information 55 or 65 is stored in each header 51 or 61, but the creator 
information 55 or 65 may be also stored in a part other than the header 51 or 61 

30 of GCip or audio frame. In this case, too, as far as the creator information is 
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recorded in each GOP or audio frame, same effects can be obtained. 

In the present embodiment, an example of data having plural 
frames managed as one group, as represented by the GOP used in MPEG 
video data, is explained, but the same effects are obtained in the video data 
5 managed in one frame unit such as motion JPEG by adding the creator 
information to each frame data. Similarly, in the data originally managed in one 
frame unit such as motion JPEG, by managing plural frames as one unit and 
adding creator information to each management unit, the same effects are also 
obtained. 

10 Further, as shown in Figs. 3 and 4, instead of adding creator 

information to each GOP of video data or each audio frame of audio data, one 
piece of creator information may be added to the entire video data or audio data 
as shown in Figs. 5A through Fig. 5C. The position at which the creator 
information is added may be beginning, middle or end of video data or audio 

15 data as shown in Figs. 5A through Fig. 5C. 

If unique identification information (ID) is inherently added to the 
video and audio data transmitted by e-mail, use of this identification information 
allows the cited data in the reply mail data to be detected. In this case, the e- 
mail processing apparatus can detect the cited data in the reply mail data 

20 without adding the creator information by managing the unique identification 
information added to the data of the transmitted mail. 

Second Embodiment 

An e-mail processing system in the present embodiment of the 
25 invention is described below. In the e-mail processing system of the present 
embodiment, the creator information is embedded as a digital watermark in the 
video and audio data transmitted as electronic mail. Fig. 6 is a block diagram of 
the system of this embodiment. 

An e-mail processing apparatus 300 of the embodiment includes a 
30 mail data creating section 301, a digital watermark embedding section 303, a 
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mail transmitting section 304, a mail receiving section 308, a digital watermark 
extracting section 309, a cited data detecting section 312, a reproduction speed 
control section 313, and a audio and video reproducing section 314. An e-mail 
replying apparatus 350 includes a mail receiving section 355, a mail editing 
5 section 356, and a mail transmitting section 357. 

In the e-mail processing apparatus 300, the mail data creating 
section 301 creates video and audio data 81 to be transmitted as video and 
audio mail. The digital watermark embedding section 303 embeds creator 
information 82 in the video and audio data 81 as a digital watermark. The mail 

10 transmitting section 304 transmits the video and audio data with embedded 
digital watermark. 

The data transmitted from the mail transmitting section 304 is 
received in the mail receiving section 355 of the e-mail replying apparatus 350. 
A reply mail is then created in the mail editing section 356 by citing a part or 

15 whole of the video and audio data. The data for reply mail created in the mail 
editing section 356 is returned to the e-mail processing apparatus 300 through 
the mail transmitting section 357. 

The mail returned from the mail transmitting section 357 is 
received in the mail receiving section 308 of the e-mail processing apparatus 

20 300, and the creator information 83 is extracted by the digital watermark 
extracting section 309, and video and audio data 84 is obtained. 

The extracted creator information 83 is compared with the creator 
information 82 of the original data in the cited data detecting section 312. 
Based on the result of comparison, data portion with the extracted creator 

25 information 83 equal to creator information 82 of the original data is extracted as 
cited data. The reproduction speed control section 313, by using the 
information extracted by the cited data detecting section 312, creates 
reproduction speed control data for the video and audio data 311. That is, the 
reproduction speed control data is created to instruct reproduction speed so that 

30 the data of the cited portion is reproduced at fast speed, while the data of other 
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than cited portion is reproduced at normal speed. According to this 
reproduction speed control data, the audio and video reproducing section 314 
reproduces the video and audio data 31 1 . 

As described above, even though the data format is converted by 
5 compression or decompression of data in the mail editing section 306 of the e- 
mail replying apparatus 350, embedding the creator information as a digital 
watermark allows the cited portion to be extracted correctly in the cited data 
detecting section 312. 

In the foregoing embodiments, any arbitrary information (for 
10 example, identification information added to each data) may be used as the 
creator information as far as the cited portion and other newly added portion 
can be distinguished in the reply mail. 

In the preceding embodiments, the reproduction speed is different 
15 between the cited data portion and newly added portion in the reply mail. 
Instead, the reproduction method may be distinct in other respects than the 
reproduction speed. For example, in the cited data portion, instead of 
reproducing the whole cited data, only a specific beginning portion of the cited 
data may be reproduced. 

20 

Industrial Applicability 

According to the invention, the reproduction method may be 
varied between the cited portion data and newly added portion data in the reply 
mail, and only the cited portion data can be reproduced at fast speed, and 
25 therefore the viewing efficiency of reply mail can be enhanced, and it is very 
useful for receiving apparatus of video or audio electronic mail. 

The invention is described so far by referring to preferred 
embodiments, but other changes, modifications and applications are apparent 
30 for those skilled in the art. Hence the invention is not limited by the illustrated 
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embodiments alone, but should be limited by the scope of the claims being 
appended. The present invention relates to Japanese Patent Application No. 
2003-429445 (filed December 25, 2003), the contents of which are incorporated 
herein by reference. 



